64 research outputs found
Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
We suggest a general oracle-based framework that captures different parallel
stochastic optimization settings described by a dependency graph, and derive
generic lower bounds in terms of this graph. We then use the framework and
derive lower bounds for several specific parallel optimization settings,
including delayed updates and parallel processing with intermittent
communication. We highlight gaps between lower and upper bounds on the oracle
complexity, and cases where the "natural" algorithms are not known to be
optimal
Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxy
We present an algorithm for minimizing an objective with hard-to-compute
gradients by using a related, easier-to-access function as a proxy. Our
algorithm is based on approximate proximal point iterations on the proxy
combined with relatively few stochastic gradients from the objective. When the
difference between the objective and the proxy is -smooth, our
algorithm guarantees convergence at a rate matching stochastic gradient descent
on a -smooth objective, which can lead to substantially better sample
efficiency. Our algorithm has many potential applications in machine learning,
and provides a principled means of leveraging synthetic data, physics
simulators, mixed public and private data, and more
- …